feat(srt): run AgentX on native srt-slurm - #3428

Step 3 of the srt-slurm migration: run all AgentX, single- and multi-node, on native srt-slurm with the design used for fixed-sequence (#3352, #2542, #3413, #3415). TileRT is out of scope.

Draft: 58 of the 62 active single-node AgentX configs now run natively, the multi-node recipes are consolidated, and MI355X DSV4 multi-node (#2555) is ported. Some smoke runs are still in flight; see Testing.

Background

• Single-node fixed-sequence points already run natively. A search-space entry names an srt-recipe, infx.srt_slurm.single_node selects the one recipe variant that matches the matrix point, and srt-slurm owns the server.
• Single-node AgentX still runs 62 active configs through 65 bash scripts that start the server and the AIPerf replay themselves. The single-node adapter rejected AgentX points.
• Multi-node AgentX already runs on srt-slurm, with its own client, benchmarks/multi_node/agentic_srt.sh.

Design

The same principles as fixed-sequence:

• one shared client for single- and multi-node;
• concurrencies come from the matrix;
• recipes carry only engine configuration;
• one recipe file per config: variations are basic override_* variants (one per point), not separate files or zip variants. Existing multi-node AgentX recipes split into one file per variation (for example qwen3.5/sglang/gb300-fp4/agentx/agg-tp2-c24-...-jid2530012.yaml) get consolidated the same way;
• engine setup that has to run in the worker container uses the recipe's setup_script, never cluster hooks.

Changes

• Shared client. benchmarks/srt_agentic.sh (moved from benchmarks/multi_node/agentic_srt.sh) serves both paths:
  • it replays one CONC point, or each point of a multi-node CONC_LIST batch with the existing drain between points;
  • behind a router frontend it reads engine metrics from each worker (SRT_*_ENDPOINTS); Dynamo is unchanged;
  • the workspace, port and multi-node flag default in the client instead of every recipe.
  • All AgentX recipes point at the new path.
• Routing. The 11 single-node launchers take the native path for any point with an srt-recipe, AgentX included. Configs without one keep the legacy script, so they move over one at a time.
• Matrix. AgentX search-space entries accept srt-recipe, and the generator passes it through as it does for fixed-sequence.
• Adapter. infx.srt_slurm.single_node accepts AgentX points and DSpark speculation:
  • it skips the fixed-sequence workload checks and requires the AgentX client;
  • it binds the AgentX runtime inputs and writes results under /logs.
  • Golden acceptance comes from synthetic_acceptance, as for multi-node AgentX.
• Artifacts. The launcher copies logs/agentic to results/, where the workflow uploads the raw replay artifacts and the power window.
• Launchers. The H200 native path sets SRT_MODEL_PATH, as the other launchers do. On B200 a model that isn't staged locally resolves as hf:<model>. GB200 and GB300 gain the native single-node path: they import the squash on a compute tray, then call launch_srt_single_node with an aarch64 srt-slurm setup (SRT_SETUP_ARCH).
• vLLM in the adapter. vLLM points validate topology as tensor-parallel-size x data-parallel-size = TP, DP attention as data-parallel-size > 1, and expert parallelism as enable-expert-parallel. Eval-only runs set max-model-len.
• Client-side chat template. Recipes set AIPERF_APPLY_CHAT_TEMPLATE: 'true' where the legacy script passed --apply-chat-template to the replay.
• Recipe fixes found by validating every active multi-node AgentX recipe against the pinned srt-slurm:
  • 6 Kimi-K3 vLLM recipes used the old provider/default_frequency telemetry fields; they now use collect_interval_ms: 1000;
  • one GLM-5.2 H200 recipe pinned collector_join_timeout_seconds: 10, which the validator rejects; srt-slurm now derives it.
• Ported configs (single-node SGLang). dsv41flash-fp4-h200-sglang-agentic-dspark, qwen3.5-fp8-h200-sglang-agentic-mtp, qwen3.8next-fp8-h200-sglang-agentic-mtp, and DSV4.1 Flash DSpark on B200, B300, GB200 and GB300 (dsv41flash-fp4-{b200,b300,gb200,gb300}-sglang-agentic-dspark). The first is described below; the others follow the same pattern. The adapter also checks KV_OFFLOADING and TOTAL_CPU_DRAM_GB when a variant declares them, which is how the Qwen3.5 GPU-KV and HiCache variants are selected.
• Pilot. dsv41flash-fp4-h200-sglang-agentic-dspark (TP4 and TP8) moves to dsv41flash/sglang/h200-fp4-mtp/agentic.yaml, translated from dsv41flash_fp4_h200_sglang_mtp.sh.
  • One basic override per (TP, concurrency) point sets admission (2x CONC, capped at the decode graph batch) and the warmup grace period.
  • The measured H200 tilings install through setup_script.

Status of single-node AgentX configs

• Native (58). Every search-space entry names an srt-recipe and binds through the adapter at every matrix point (perf and eval), except the legacy entries listed next.
• Entries kept on legacy scripts:
  • ATOM LMCache entries of glm5.2-fp4-mi355x-atom (DCP4), kimik3-fp4-mi355x-atom (DCP8) and minimaxm3-fp4-mi355x-atom: srtctl's ATOM backend reserves kv-transfer-config, so an aggregate worker can't set the LMCache connector without an srt-slurm patch.
  • kimik3-fp4-mi355x-vllm and minimaxm3-fp4-mi355x-vllm: their nightly images were pruned from Docker Hub. The legacy path still runs from node-local squashes. The native path can use a squash staged on /it-share (the launcher already looks there); the recipes are ready.
• Not ported: qwen3.5-{fp4,fp8}-b300-sglang-agentic-power-ab target cluster:b300-nv, which has no runners.
• Blocked by infrastructure, wired natively:
  • the 4 MI325X configs: every MI325X job, main included, fails in the workflow's uv python install (File exists at the runner's ~/.cache/uv);
  • minimaxm3-fp4-{b200,b300}-vllm: image pruned, no squash cache, and the legacy path fails the same way.

Testing

All runs use agentx-fast, single concurrency, --no-evals.

Single-node, native (new path)

Multi-node, existing srt-slurm path (shared client)

Single-node, ported batches. Each ported config was smoke-tested on its cluster with agentx-fast at points covering each variant regime, and at least one eval-only point per engine family. Runs are listed in the batch comments on this PR.

• B200 SGLang/TRT, B300 SGLang/TRT, H100/H200 SGLang and vLLM, B200/B300/GB200/GB300 vLLM, MI300X/MI355X vLLM, MI355X SGLang/ATOM.
• Fixes found by these runs (all in this PR):
  • B300 jobs got srtctl's default one-hour --time; the profile now takes the workflow limit.
  • Models not staged on every B300 node now load from the shared model root.
  • GB200/GB300 cold loads take about 30 minutes; the recipes get a two-hour health budget.
  • Single-node AgentX eval-only failed on every engine (a MAX_MODEL_LEN requirement, then double artifact staging); it now evaluates at native context with the workflow's framework, and artifacts upload.
  • H100 per-runner uv caches, MI355X shared HF cache mounts, and a squash staged on /it-share for the MI355X native path.
• Multi-node consolidation: 122 per-variation recipes became 29 files with override_* variants; every selector resolves identically to the file it replaced (0 diffs), and smoke passed on B200, B300, GB200 and GB300.
• MI355X DSV4 multi-node (UMBP): c4, c16 and c192 passed (36158061438); no power data, as on main.

Known issues

• H200 DSV4/GLM-5.2 AgentX (pre-existing). These recipes enable DCGM power telemetry and place etcd/NATS on a dedicated node, and the pinned srt-slurm rejects that combination (telemetry requires infra.etcd_nats_dedicated_node: false). RECIPES.md already records this as a power compatibility blocker that should not be fixed by changing topology, so this PR leaves the placement as is.
• Kimi-K3 H200 vLLM. The replay ran and wrote valid results (0/5 errors), then aiperf failed ProfileMetricCoverageError: at c1 with the 20-minute agentx-fast profile, only 5 to 7 long requests (median E2E 113 s) completed and none ran into the last 60 s. A full-duration rerun is pending to separate that from a server stall.

Remaining work

• Port the remaining single-node AgentX configs (in progress, batched by cluster and engine: B200, B300, NVIDIA vLLM, H100/H200, MI355X SGLang/ATOM, AMD vLLM).
• Consolidate multi-node AgentX recipes split into one file per variation into one file per config with basic overrides (in progress).
• MI355X multi-node DSV4 (Port MI355X DeepSeek V4 AgentX DSPARK to AMD-enabled srt-slurm #2555, UMBP linker) is ported here and under test; then remove the legacy scripts and the amd_utils AgentX path.
• qwen3.5-{fp4,fp8}-b300-sglang-agentic-power-ab target cluster:b300-nv, which has no runners in configs/runners.yaml; they are left as is.
• TileRT follows in a separate PR stacked on this one.